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Corporate and educational settings increasingly require decision-making, 
problem-solving and other complex cognitive skills to handle ill-structured, or 
heuristic, tasks, but the growing need for heuristic task expertise has outpaced 
the refinement of task analysis methods for heuristic expertise. The Heuristic 
Task Analysis Method was applied to three settings to generate improvements 
and more detailed guidance, and to identify variations in the method for different 
situations. The three settings were group counseling, tutoring on writing skills, 
and selecting artwork for a product line. The formative research methodology 
was used to test the method and generate improvements. The three studies 
produced some common and some unique findings and recommendations. A 
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INTRODUCTION 

As our society in general and the workplace in particular become more complex, 
we are finding that a greater number of the activities that people undertake are 
relatively more heuristic in nature than ever before. Whether in K-12 education, 
higher education, corporate training, or any other context, to help people learn 
the heuristic elements of an expert's know-how, we must be able to identify those 
heuristics. 

In analyzing heuristics, we find it helpful to think in terms of two major kinds of 
expertise — domain and task expertise. 

Task expertise relates to the learner becoming an expert in a 
specific task, such as managing a project, selling a product, or 
writing an annual plan. Domain expertise relates to the learner 
becoming an expert in a body of subject matter not tied to any 
specific task, such as economics, electronics, or physics (but often 
relevant to many tasks). (Reigeluth, 1999, p. 435). 

Both procedural and declarative knowledge are important elements of both kinds 
of expertise. In this research, we focus on task expertise. 

For task expertise, we find it helpful to think in terms of two major kinds of tasks: 
procedural and heuristic. Procedural tasks are "tasks for which experts use a set 
of steps, mental and/or physical, to decide what to do when, such as a high 
school course on mathematics or a corporate training program on installing a 
piece of equipment for a customer. Heuristic tasks are "tasks for which experts 
use causal models — interrelated sets of principles and/or guidelines — to decide 
what to do when, such as a high school course on thinking skills or a or a 
corporate training program on management skills." (Reigeluth, 1999, p. 435). 

The distinction between procedural and heuristic tasks is similar to the distinction 
between well structured and ill structured domains (Fredericksen, 1984; Resnick, 
1988; Simon, 1973). In reality most tasks are neither purely procedural nor 
purely heuristic, but some combination of the two. We have relatively powerful 
methodologies for analyzing the expertise that underlies procedural tasks (i.e. the 
mental and physical steps upon which an expert relies). But we do not have 
good methodologies for analyzing the expertise that underlies heuristic tasks. 
This situation is exacerbated by the reality that heuristic knowledge is frequently 
tacit rather than explicit — that is, experts are often unaware of the heuristics that 
guide their performance. Therefore, there is a strong need to develop task 
analysis methodologies for identifying the knowledge that underlies heuristic 
tasks. This paper briefly reviews literature related to heuristic task analysis. 
Then it describes three research studies that have been conducted to improve 
one of those methodologies. 
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REVIEW OF THE LITERATURE 

In reviewing the literature, we identified knowledge elicitation, analysis, and 
representation as three important aspects of heuristic task analysis. In the 
following section, we will examine various techniques that are frequently used in 
this field, from traditional task analyses to heuristic task analysis, on the basis of 
guidance provided for those three aspects. We found that many of them deal 
with procedural tasks rather than heuristic tasks, and those that deal with 
heuristic tasks focus primarily on how to elicit experts' knowledge. Furthermore, 
it is difficult to find any research that provides guidelines on how to analyze and 
represent such knowledge. 

Knowledge Elicitation 

Knowledge elicitation refers to the extraction of domain-relevant knowledge 
directly from a human expert using various techniques (Jones, Miles, & Read, 
1996). Research on knowledge elicitation has been conducted mostly in relation 
to expert systems that use expert knowledge to perform complex problem-solving 
and decision-making processes, and many studies reported that knowledge 
acquisition has continued to be the major bottleneck in the process of building 
knowledge-based systems (Cooke, 1994; Wood & Ford, 1993). 

Based on the belief that the better the data gathered in elicitation, the better the 
resulting model of expert knowledge, many researchers have been interested in 
identifying relevant knowledge elicitation techniques. Some of them have 
focused on the type of knowledge elicited by the technique with an assumption 
that different elicitation methods tap different types of knowledge (Kitto & Boose, 
1987; Kitto & Boose, 1989; Wielinga, Schreiber, & Breuker, 1992). Others have 
categorized techniques according to the stage in the elicitation process in which 
the techniques are employed (Olson & Biolsi, 1991; Shaw & Woodward, 1989). 
There are also research works that organize the knowledge elicitation techniques 
based on the mechanics of the techniques themselves (Jones et al., 1996; 
Tomlinson & Johnson, 1994). 

One of the most comprehensive reviews of knowledge elicitation techniques was 
done by Cooke (1994). She categorized various knowledge elicitation methods 
into three families on the basis of methodological similarity: observations and 
interviews, process tracing, and conceptual techniques. Table 1 summarizes key 
characteristics, advantages, and disadvantages of each family of methods based 
on previous studies. 



Insert Table 1 about here 



Observations and interviews. This family involves rather direct methods of 
watching experts and/or talking with them, and is excellent for forming an initial 
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conceptualization of the problem domain. These are the most frequently used 
techniques, but they are relatively informal, with much of the specification of 
methods and analyses left to the elicitor's intuition (Bell & Hardiman, 1989; 
Brenner, Brown, & Canter, 1985; Cooke, 1994; Cordingley, 1989; Forsythe & 
Buchanan, 1989). Observation can be used to identify problem-solving 
strategies that are not consciously accessible to an expert and to verify an 
expert’s description of what he or she actually does without interfering with the 
expert’s task performance and environment. However, it is often difficult to 
interpret the observation data, especially for heuristic (or complex cognitive) 
tasks, and there may be some influence of the analyst’s presence on the expert’s 
task performance. Besides that, for tasks in which real-time process cannot be 
controlled, detailed observation is problematic and often impossible (Clarke, 
1987). Interviews are usually retrospective in that the expert is required to 
retrieve information based on past experiences. There are two types of 
interviews: unstructured and structured. The former is a free-form interview in 
which neither the content nor the sequencing of the interview topic is 
predetermined (Cooke, 1994; Hoffman, 1987; Kidd & Cooper, 1985; Weiss & 
Kulikowski, 1984; Welbank, 1990). 

Process Tracing. The second family of techniques, process tracing, includes 
protocol analysis, content analysis, discourse analysis, and decision analysis. 
These techniques are performed concurrently with the expert's task performance, 
record pre-specified types of data (e.g. verbal reports, eye movements, actions), 
and use those data to make inferences about the cognitive processes underlying 
task performance (Cooke, 1994). Decision analysis is somewhat different from 
the rest of this family in that it uses formal statistical methods to provide 
quantitative information about decisions made by the experts during the task 
performance. 

Conceptual techniques. This family of techniques entails the extraction of 
domain concepts and the structure or interrelations among concepts. Repertory 
grid, concept listing, rating, and ranking belong to this family, and they tend to be 
indirect and require less introspection and verbalization than interviews or other 
types of verbal report techniques (Cooke, 1994). 

Each family of techniques has its own advantages and disadvantages, and there 
seems to be no one single family that is suited for every purpose. Indeed, the 
studies on various knowledge elicitation techniques have emphasized the notion 
that one should not rely on any single method (Cooke, 1994; Hoffman, 1987; 
Shadbolt & Burton, 1989). It seems that the major problem with knowledge 
elicitation techniques is not a shortage of methods but a lack of advice on how to 
select and apply those methods depending on the type of task and contextual 
constraints. In other words, there has been the general tendency to "name 
methods without providing much information on how to apply them" (Forsythe & 
Buchanan, 1989) and "an amazing lack of evaluative information on the methods, 
particularly in the form of empirical data" (Cooke, 1994). 
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Knowledge Analysis 

Once relevant data are extracted, the next step is to interpret the data. Recent 
studies have often used computer programs to automate the analysis process, 
representing a shift away from knowledge analysis in which human analysts are 
involved toward more 'non-human' based approaches (Olson, Mechitov, & 
Moshkovich, 1995; Sommer, Morik, Andre, & Uszynski, 1994). For example, 
Benbenishty (1992) reported the use of a computer program for structured 
interviews in which the program constantly analyzed the expert's responses, 
finding inadequately covered areas, identifying apparent contradictions that 
should be clarified, and eliciting needed refinements (thus overlapping with the 
knowledge elicitation phase described earlier). He also described how 
sophisticated programs could be used in protocol analysis, partitioning 
transcribed protocols of the expert into segments on the basis of speech pauses. 

However, there is little guidance concerning the analysis of the data generated 
by interviews and observations, and without a systematic framework for 
analyzing such data, the reliability and validity of data gathered from these 
knowledge elicitation techniques have often been questioned by researchers 
(Cooke, 1994; Hoffman, Crandall, & Shadbolt, 1998). 

Knowledge Representation 

As mentioned earlier, knowledge elicitation methods have been mostly used with 
a fairly narrow set of applications in mind, primarily the development of expert 
systems and artificial intelligence (Al) systems. Expert systems are computer 
systems designed to tackle problems normally thought to require the intervention 
of human experts, and the overall goal of knowledge elicitation is to externalize 
knowledge in a form that can be implemented in such systems (Cooke, 1994; 
Hoffman et al., 1998; LaFrance, 1992). Not surprisingly, a central problem for 
researchers has been to find ways to represent expert knowledge in a manner 
that computers can use. Some of the frequently used knowledge representation 
schemes include semantic networks, augmented transition networks, bites, 
frames, scripts, and grammar. They have been selectively used to construct 
systems that hierarchically organize information for tutoring, simulation, and 
parsing of natural language, depending on the type of problem being addressed 
(Nelson, 1989; Wenger, 1987). 

Other types of knowledge representation that are not directly related to computer 
systems are mostly from traditional task analysis methods such as functional flow 
analysis, information flow analysis, and interaction analysis. Task analyses have 
been commonly used in human factors and industrial engineering applications, 
and have provided a means of representing a task using information that has 
been elicited through informal observations and interviews. Functional flow 
analysis involves the creation of diagrams that display the primary system 
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functions and subfunctions in a sequence, and information flow analysis 
displays necessary information and decisions to carry out the system functions in 
a flowchart. In interaction analysis, major constraints that are imposed on the 
system are identified and presented in a graphical format (Cooke, 1994; 
Jonassen, Hannum, & Tessmer, 1989; McGraw & Harbison-Briggs, 1989). 

It appears that different types of knowledge require different types of 
representation (Nelson, 1989; Wenger, 1987). A method that can best represent 
procedural knowledge might not be equally effective for heuristic knowledge, but 
there is a gap in previous studies in advising how to represent heuristic 
knowledge. 

Knowledge representation is often determined by the purpose of its application. 
Building an expert system has been a prevalent reason for conducting task 
analyses, and research on knowledge representation has been driven by that 
purpose. As a consequence, there is a lack of research on knowledge 
representations to inform instructional designers on how to utilize heuristic 
knowledge in developing an instructional system. 

In the following section, we will introduce task analysis methodologies that are 
more focused on heuristic tasks. 

Heuristic Task Analysis Methodologies 

Olson and Biolsi (1991) present, compare, and contrast a variety of methods for 
capturing and representing expert knowledge. They summarize the various 
methods they present as passing through “four distinct stages": 

1 . elicit behavior from the subject; 

2. summarize the behaviors that can be analyzed from different algorithmic 
perspectives, 

3. analyze the data using one of a variety of methods; 

4. display the results. (Olson & Biolsi, 1991). 

For example, elicitation methods (step 1 above) include interviews, think aloud, 
grouping items, sorting items and others. Summary methods (step 2 above) 
include transcribing and using proximity matrices. 

Cognitive Task Analysis (CTA) 

Cognitive task analysis is a process of identifying the performance components, 
knowledge structures, and metacognitive knowledge underlying a task (Dehoney, 
1995). To elicit expert knowledge, researchers tried various techniques that 
were first used by cognitive scientists studying human cognition in a lab setting, 
such as interview, focused discussion, protocol analysis, and simulations, as well 
as on-site observation (Dehoney, 1995; Black, Dimaraki, Esselstyn, & Flanagan, 
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1995). Typically CTA involves either participants working on domain-free 
problems or knowledge workers who solve difficult problems for a living (e.g., 
aviators or X-ray technicians). 

Dehoney (1995) suggested two phases in conducting CTA: (1) to define 
competent performance of the task, and (2) to derive a performance model that 
specifies how practitioners actually perform the task. The goal of the first phase 
is to identify what people must be able to do to accomplish the task at a 
satisfactory level, what kinds of problems they must solve, what must they know 
and how must they use this knowledge to solve problems. Based on the analysis 
of the competency model, the second phase of analysis intends to create a 
performance model to improve people's performance. However, whether or not 
an expert's domain-specific skills and knowledge captured in one model can be 
transferred to other domains remains a controversial issue among researchers in 
evaluating the effectiveness of such performance models. 
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Integrated Task Analysis Model (ITAM) 

Ryder and Redding (1993) have proposed a task analysis model to combine 
cognitive task analysis with traditional behavioral task analysis models. They 
argue that each type of method is incomplete and that they should be used 
together to adequately represent many tasks. 

Ryder and Redding (1993) divide expertise into skills (procedural knowledge), 
knowledge (declarative knowledge), and mental models. They define mental 
models as “functional abstractions about a job or job task which provide a 
deductive framework for problem solving” (p. 85). Thus, mental models may 
include a variety of conceptual, procedural, decision-making, heuristic and 
strategic knowledge. Ryder and Redding describe four analysis techniques from 
the literature that they recommend for developing the mental models: 
psychological scaling, protocol analysis, cognitive interviewing, and cognitive 
modeling. 

Psychological scaling (Ryder & Redding, 1993) involves statistical techniques to 
determine the organization of conceptual knowledge and is sometimes used to 
examine expert skill that is automatic and unavailable for conscious 
introspection. Protocol analysis (Ryder & Redding, 1993) has experts think aloud 
while performing a task and is useful for developing heuristic knowledge and 
mental models. Cognitive interviewing uses structured or semi-structured 
questions “based a cognitive theory of expertise and a desired framework for 
representing or using the results” (Ryder & Redding, 1993, p. 79). This method is 
also useful for developing heuristic knowledge and mental models. Cognitive 
modeling is a process in which the analyst constructs a model of “the cognitive 
processes that accurately describe or predict human performance” (Ryder & 
Redding, 1993, p. 80). Cognitive modeling is a time-consuming process that is 
applied to complex cognitive domains. 

Expertise Analysis 

Martin Backler describes a procedure for conducting an expertise analysis that 
was presented at the 1990 NSPI Conference in Toronto, Canada. Expertise 
Analysis is a method for using cognitive science tools to uncover how experts 
solve problems (Backler, 1990). The procedure is as follows: 

1 . Select a suitable content domain. 

2. Identify appropriate experts. 

3. Collect suitable problem solving material. 

4. Work with your experts. 

5. Analyze the problems used. 

6. Build a model of how the experts solve problems. 

7. Build a model of the knowledge used. 

8. Evaluate the model. 
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Backler gives more specific details about each of the steps listed above, 
including selecting the proper domain to be analyzed, selecting the most suitable 
experts to be analyzed, and specifying researchers to use the protocol analysis 
technique. He also recommends using expert reviews to validate the models that 
represent the expert’s problem solving process and knowledge. 

Creating Organizational Knowledge from Tacit Knowledge 

Nonaka (1994) describes a systemic process for making tacit knowledge explicit. 
He breaks tacit knowledge into cognitive and technical elements. The technical 
element includes specific know-how, crafts and skills. The cognitive element 
includes working mental models, such as schemata, paradigms, beliefs, and 
viewpoints. Tacit knowledge is created by individuals and the organization can 
support individual creativity and provide an environment for individuals to interact 
with others to create explicit, or organizational, knowledge. 

Nonaka proposes that “tacit knowledge may be transformed into explicit 
knowledge by (1) recognizing contradictions through metaphor, and (2) resolving 
them through analogy. Explicit knowledge represents a model within which 
contradictions are resolved and concepts become transferable through 
consistent and systematic logic” (p. 21). Nonaka asserts that explicit knowledge 
is created and legitimized through social interaction, which can occur at the level 
of informal groups, the entire organization, or even between organizations. 
Nonaka suggests some ways of promoting social interaction to foster explicit 
knowledge creation. 

The Heuristic Task Analysis Method of Elaboration Theory 

The Elaboration Theory (Reigeluth, 1999; in press) offers the Heuristic Task 
Analysis (HTA) method as part of its Simplifying Conditions Method (SCM) for 
task analysis. This more general method of task analysis is conducted by asking 
the question, "What is the simplest version of the task that an expert has ever 
performed?" and "What is the next simplest version?" and so forth. As each 
version is identified, its place in the learning sequence is simultaneously 
determined. Therefore, the SCM task analysis method is an integral part of the 
method for designing an instructional sequence. Furthermore, since most tasks 
have a combination of procedural and heuristic elements, the procedural and 
heuristic task analysis methods are integrated into a single process. For these 
reasons and because this method offers a fair amount of guidance for the task 
analysis process, we chose this method for our research. Hence, it is described 
in greater detail next. 

The SCM's Heuristic Task Analysis Method 
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The following are some details on the SCM's heuristic task analysis method. 

They are an elaboration of the process described by Reigeluth (1999). 

Phase I. Prepare for Analysis and Design 

1. Prepare. Lay the groundwork for your analysis and design. 

1 . 1 Establish rapport with a task expert. 

1 .2 Explain the analysis process you will be using. 

1 .3 Ask the task expert about the nature of the task in general. 

1 .4 Identify the characteristics of the learners in general. 

1 .5 Identify the delivery constraints of the task in general. 

Phase II. Identify the First Learning Episode 

2. Identify the simplest version. Help the task expert to identify the simplest version of 
the task that is fairly representative of the task as a whole, and to describe the conditions 
that distinguish that version from all other versions. 

• You may want to use some other criteria in addition to simple and 
representative, such as common (how frequently performed the version of the 
task is) and safe (how much risk there is to the learner and/or the equipment). 

• Ask the task expert to recall the simplest case she or he has ever seen. The 
simplest version will be a class of similar cases. Then check to see how 
representative it is of the task as a whole. 

• It may be helpful to start by identifying some of the major versions of the task 
and the conditions that distinguish when one version is appropriate versus 
another. 

• Thinking of different conditions helps to identify versions, and thinking of 
different versions helps to identify conditions. Hence, it is wise to do both 
simultaneously (or alternately). 

• There is no single right version to choose as the "simplest." It is usually a 
matter of trade-offs. The very simplest version of the task is usually not very 
representative of the task as a whole. The more representative the simple 
version can be, the better, for it provides a more useful schema to which 
learners can relate subsequent versions. 

• It may be wise to go through this process with several task experts before 
going on to Step 3. You may find it necessary to take steps to resolve 
differences of opinion about which is the best “simplest version” to use. 

3. Analyze the organizing content. Analyze the organizing content (mostly heuristics 
and descriptive theories) for this version of the task. 

3. 1 Ask the task expert to think of one specific performance of the task to 
analyze, or to videotape a performance for you to review with the expert during 
the analysis. 

3.2 Use a top-down approach to analyzing the content (the knowledge upon 
which the expert’s performance is based). In other words, start by identifying the 
general categories of knowledge that an expert uses, then proceed to analyze 
each. 

• Ask the task expert: 

a) to describe each decision that the task expert made, 
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b) to identify the kinds of knowledge that the task expert drew 
upon to make the decision, and 

c) to describe the specific knowledge within each kind of 
knowledge that the task expert used. 

The kinds of knowledge are likely to include: 

- steps (procedural knowledge), 

- guidelines or rules of thumb (heuristic knowledge), 

- explanatory models (which explain why the guidelines work), 

- descriptive models (which describe the phenomena with 
which the task expert interacts), and 

- metacognitive/decision rules (which the task expert uses to 
decide which steps, guidelines, and descriptive knowledge, to 
use when). 

It is generally helpful to start by asking the task expert if there are any 
steps or phases of activities that are always performed for this version 
of the task. If so, perform a procedural task analysis to identify the 
sequence of steps and to see if any of those steps can be broken 
down into substeps, but those substeps must be ones that an expert 
thinks of and uses routinely in performing that version of the task. 

For guidelines, use the following process (See the example in Table 
2 below.): 

1 . Identify the goals for this specific performance of the task 
under its conditions. 

2. Identify all the important considerations for attaining each 
goal. Considerations are the major categories of causal factors 
that influence performance of the task. If there are a lot of 
causal factors for a consideration, it is useful to identify 
subconsiderations for it. 

3. Identify all the important causal factors for each consideration 
(or subconsideration). 

4. Analyze each causal factor to identify all guidelines 
(prescriptive principles or “rules of thumb") that an expert uses 
to account for this consideration. 



Insert Table 2 about here 



For explanatory models, use the following process: 

1 . For each guideline, ask the task expert for the reasons why 
s/he believes it works. 

2. For interrelated guidelines, you are likely to identify a set of 
related reasons that constitute a causal model or models. Be 
sure to look for multiple causes for each effect and multiple 
effects for each cause. Also look for chains of causes and 
effects, and explore probabilities for each causal factor to have 
each effect. 

For descriptive models, use the following process: 

1 . Ask the task expert what phenomena influenced this particular 
performance of the task. Try to identify all causal relationships 
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that characterized those phenomena. Be sure to look for 
multiple causes for each effect and multiple effects for each 
cause. Also look for chains of causes and effects, and explore 
probabilities for each causal factor to have each effect. 

• For metacognitive/decision rules, use the following process: 

1 . Find out what rules the task expert used to decide when to use 
which steps, guidelines, and descriptive models during the 
specific performance of the task being analyzed. 

• It is wise to query the task expert about any of these kinds of 
knowledge that are not initially described to you for each decision the 
task expert made in this specific performance of the task. 

3.3 Ask the task expert to think of similar performances of the task that 
constitute a single version of the task. Use each such performance to broaden 
the steps, guidelines, explanatory models, descriptive models, and 
metacognitive/decision rules so that they represent the knowledge bases the task 
expert uses to deal with all performances for that version of the task. 

3.4 If time and resources permit, find a second task expert with whom to repeat 
this entire process (Steps 1 - 3.3) to identify any alternative views of the task and 
the knowledge that underlies its performance. It may even be wise to repeat this 
process with several more task experts. And you may want to go back and ask 
each task expert what s/he thinks about the perspectives of the other task 
experts, in an effort to reconcile conflicts and select among alternative ways of 
thinking about and performing the task. 

The HTA method has not been rigorously tested and therefore is in need of 
further research. However, the most important research issue is not the validity 
of the HTA method, for, like most methods, it is likely to work some times but not 
always. Rather, given the immaturity of our knowledge about how to analyze 
heuristic tasks, what is needed most at this point is developmental research — 
research that is intended to further develop and improve the method. Therefore, 
our research question is, "In what ways can the HTA method possibly be 
improved?" To answer this question, it is also necessary to find out what parts of 
the method are working well and what parts are not working so well. 

Furthermore, to improve the HTA method, it will likely be important (1) to change 
parts of the method, (2) to provide more detailed guidance about how to 
accomplish particular parts of the method, and (3) to identify variations in the 
method for different situations, such as different kinds of tasks or even different 
kinds of task experts. 

To answer these research questions, we conducted a series of three 
developmental research studies. These are described next, followed by some 
general conclusions. 
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Study 1: The HTA Method Applied to Group Counseling 
Methods 

The purpose of this study was to improve the HTA method and guidance for use 
of that method. Thus, the formative research methodology was adopted with 
emphasis on exploring how the HTA method can be improved when applied to 
group counseling. 

Formative research is a kind of developmental research or action research that is 
intended to improve design theory (Reigeluth & Frick, 1999). In contrast to 
research on descriptive theory, which emphasizes validity or how well the 
description matches the reality of "what is," research on design theory is more 
concerned with preferability, the extent to which one method is better than other 
methods for achieving certain goals under certain circumstances. By creating or 
identifying an instance of a design theory and collecting formative data to 
improve that instance, one may develop a better understanding of how the theory 
works in the field and thus be able to propose improvements for the theory, which 
of course would need to undergo further testing. 

Formative research can be used for improving an existing theory or developing a 
new theory, and in either case, it is a type of case study approach. Case studies 
can be classified as designed cases or naturalistic cases, depending on whether 
or not the situation under investigation is manipulated by the researcher or just 
observed. 

This study used a designed case to improve an existing theory, because the HTA 
process was intentionally instantiated and manipulated by the researcher. It 
followed the steps suggested by Reigeluth and Frick (1999): 

1 . Select a design theory: The theory under investigation was the HTA 
method. 

2. Design an instance of the theory: The design instance in this study was 
the use of the HTA method to analyze "group counseling," which is a 
heuristic task. 

3. Collect and analyze formative data on the instance: To identify problems 
and improve the HTA method, formative data were collected through 
interviews with task experts. 

4. Revise the instance: This application of the HTA method was revised 
based on the data collected about how to improve it. 

5. Repeat the data collection and revision cycle: The process of formative 
data collection and analysis was iterative. This study went through three 
revision cycles with several rounds of data collection and analysis. 

6. Offer tentative revisions for the theory: The researcher came up with a set 
of tentative recommendations to improve the HTA method, based on the 
improvements made to the instance. 
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Task 

Group counseling was selected as the task to which to apply the HTA method. 
The nature of group counseling varies depending on the nature of the group that 
the leader facilitates, and this study focused on the counseling process for 
"personal growth" groups. The personal growth group is intended to help 
relatively healthy people to explore personal issues with which most people 
struggle at various transition periods in life and thus to function better on an 
interpersonal level (Stockton, Morran, & Nitza, in press). 

Group counseling is a combination task with both procedural and heuristic 
elements. It is procedural in that the activities of the group leader are largely 
determined by the stages that a group goes through (i.e., forming, norming, 
storming, and performing), and the leader cannot help the group to progress to 
the next stage without performing certain tasks (steps) at each stage. However, 
at a deeper level of analysis, the knowledge required for the leader to decide 
when and how to intervene is not a set of steps but a set of guidelines and 
principles, which is heuristic knowledge. 

Participants 

This study involved three participants as task experts in group counseling (see 
Table 3). Expert #1 was a professor in the Counseling Department and the most 
experienced of the three task experts. The other two were doctoral students in 
the same department. They were all experienced in conducting "personal 
growth" group counseling, and their expertise ranged from three to more than 20 
years. 



Insert Table 3 about here 



Data Collection Methods 

Interview. Semi-structured in-person interviews were used as the primary data 
collection method. The second author conducted six interviews between 
September and November 1999. Each interview took 30-90 minutes and was 
audio taped for analysis. 

The purpose of the interviews was to find ways to improve the HTA method for 
eliciting, analyzing, and representing the expert's heuristic knowledge for 
performing the task of "personal growth" group counseling. The investigator 
played two roles, one as a task analyst proficient in the HTA method and the 
other as researcher searching for ways to improve the HTA method. As task 
analyst, the investigator developed a set of interview questions (see Appendix A) 
for the interview based on the HTA method, but as researcher the investigator 
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was not restricted to the predefined questions. Depending on the expert's 
response, the researcher revised the HTA method for the next interview. Thus, 
the overall interview process was flexible and reflective in nature. 

The first interview followed the sequence of the HTA method described earlier in 
this report. It started with asking questions related to the nature of the task in 
general and then moved into questions related to identifying the simplest version 
of the task, asking the task expert to recall a specific instance (or case) that fits 
into the simplest version. For the second interview, several minor modifications 
were made based on the previous interview, and once the experts agreed on the 
boundaries of the simplest version of the task, the focus of the interview moved 
into identifying specific decisions made and the underlying principles and models 
that the experts used. 

Besides changes in the type of questions and the sequence of those questions, 
each interview used a little bit different technique from the previous ones. For 
example, after the fourth interview, the researcher (in the role of analyst) 
provided index cards (Davies, 1997) summarizing the critical incidents as a 
reference to help the experts recall specific heuristic knowledge already 
identified, whereas the first three interviews did not have any such reference 
materials. 

Videotapes. Because of the confidential nature of group counseling, direct 
observation or videotaping of an expert's task performance (as called for by the 
HTA method) was not allowed. Instead, the researcher (as analyst) used a 
series of instructional videos that simulated group counseling sessions for 
beginning group leaders, to provide the analysts with a concrete case 1 . The 
video series was developed by one of the experts who participated in this study, 
and it was composed of three videos, each of which covers different stages in 
group counseling. This study focused on the first three episodes dealing with the 
beginning stage of group counseling, and it used them as if they were recordings 
of the experts' task performance 2 . Based on the video, each expert was asked to 
identify with one of the co-leaders in the video series and explain the decisions of 
the co-leaders from that person's perspective. 

Data Analysis and Interpretation Methods 

The HTA method is an iterative process: finishing the first round of HTA is not the 
end of the study but the beginning of the second round of HTA; and the end of 



' The experts could add more details to or delete certain parts, but mostly they accepted 
the scenario from the video without modification. 

2 This was possible because all three experts who participated in this study were familiar 
with this video series. Expert #1 was the author of this video series, and the other two 
experts had taken classes from expert #1 and had studied the videos as class 
materials. 
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the second round is, again, the beginning of the third round; and so on. The 
investigator went through two rounds of HTA in this study. 

Triangulation. To enhance the thoroughness of the data, this study involved 
three experts as data sources. Each of them played somewhat different roles 
during the interviews. During each round of data collection, expert #2 provided 
the initial structure of the task setting and knowledge base. Then experts #1 and 
#3 reviewed the knowledge elements, verified them, and provide additional 
information. There were a few times when the three experts did not agree with 
one another. In such cases, expert #1's judgment was accepted, as he was the 
most experienced group counselor. 

Member checks. After each interview with an expert, the researcher transcribed 
the interview and took the summary and interpretations to the next interview for 
review. Through this process, the experts corrected errors or misconceptions by 
the researcher, and the researcher asked additional questions to clarify the 
information. 

Consultation. During the data collection and analysis process, the researcher 
regularly met with the other three researchers in this study and consulted them in 
designing the interview protocol and analyzing the data. 

Results and Discussion 

As mentioned earlier there were two rounds of data collection in this study. The 
first round involved initial interviews with three experts. Instead of finishing with 
one expert and then starting with another, the researcher worked with the three 
experts simultaneously (but separately) due to their time schedules. This 
situation involved some tradeoffs. It worked well in the sense that the researcher 
could get the three experts to reach consensus on the simplest version of the 
task early in the HTA process. However, communicating with all three experts 
simultaneously was not easy for the researcher, and the researcher had to spend 
most of the time during the interviews explaining to each expert the previous 
interviews with other experts. Even though the task was a common one, the 
experts still had difficulty explaining the detailed decision-making process when 
the problems to which to apply the process had originally been identified by 
another expert. Facing this problem, the researcher decided to use an existing 
instructional video series (with which all three experts were familiar) as a frame of 
reference, instead of trying to build a new scenario based on each expert's 
experience. 

The second problem the researcher encountered during the first round of HTA 
was that the researcher lacked expertise in the task of group counseling. The 
researcher found that, to be able to push the expert to further elaborate his/her 
automatized (and hence subconscious) task expertise, the analyst needed to 
speak the same language as the expert and be able to prompt when the expert 
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had difficulty in finding the right words. Without a certain level of expertise in the 
task domain, the researcher as task analyst had difficulty doing those jobs 
smoothly. The researcher tried to become familiar with the field by reading some 
introductory textbooks and articles written by the primary expert who participated 
in this study. However, learning about the field was time consuming and could 
not be satisfactorily achieved within the time constraints. 

The third problem was related to the difficulty of categorizing the types of 
knowledge underlying each decision made by the expert during the task analysis. 
Even though the researcher explained the differences among the various types 
of knowledge according to the HTA, such as guidelines and decision rules, often 
the experts had difficulties in distinguishing those concepts. Not being familiar 
with the field, the researcher had difficulty labeling the type of knowledge 
underlying certain decisions. The purpose of identifying the five types of 
knowledge specified by the HTA was to make sure that the expert did not 
overlook one of the important types of knowledge, but the benefits of 
distinguishing among the types seemed to not be worth the extra time required in 
this case. 

The second round of HTA incorporated some new methods to deal with the 
problems found in the first round. First, the researcher summarized key incidents 
from the video series on index cards and used them as a reference during the 
interviews with the experts. This was very helpful in three ways: (1) it helped the 
experts to recall details about the task performance process; (2) it helped both 
analyst and experts to see the flow of the task performance process and get back 
on track when the experts got off-task; and (3) it saved a lot of time in revisiting 
previous points. One expert commented that the index cards forced him to be 
more precise during the review and revision process. 

Second, the researcher as analyst used a bottom-up approach (identifying 
knowledge first, then categorizing it as to type) rather than the top-down 
approach (identifying knowledge within each type) suggested by the HTA 
method. In the first round, the experts had difficulties in identifying the kind of 
knowledge underlying each decision and spent a lot of time on that. Thus, in the 
second round the analyst asked the experts to describe all knowledge related to 
performing the task first, regardless of the type of knowledge, trying not to 
interrupt their cognitive process. Later the analyst categorized each piece of 
knowledge and then asked the experts to verify them. This bottom-up approach 
seemed to be more efficient than the top-down approach, especially for the less 
experienced experts. 

Based on the findings of this study, the following changes are proposed as 
possible improvements and described in detail below: (1) incorporate various 
interview and observation techniques into the HTA process; (2) provide different 
guidelines for analysts with different levels of task expertise; (3) provide different 
guidelines for working with task experts with different levels of expertise; and (4) 
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provide reference material during the interview with task experts. Each of these 
is described next (see Table 4). 



Insert Table 4 about here 



1. Incorporate various interview and observation techniques. In-person 
interview appeared to be an effective way to elicit the expert's covert heuristics 
and make them explicit by verbal explanation. However, the expert was not 
always able to verbalize his or her own thinking process. Thus, the analyst 
needed to develop his or her own sense of "heuristics" through observing the 
expert's task performance or reviewing reference materials to fill in the gap 
between what was said and what remained unsaid. Especially as an analyst with 
little expertise in the task domain, direct or indirect observations helped the 
analyst to understand the workflow and task. Unfortunately, for a task that 
involves interaction with other people or requires confidentiality, such as group 
counseling in this study, direct observational techniques were not appropriate, 
and the analyst needed to find alternatives, such as simulated videotapes or 
staged cases. Given these considerations, specific suggestions for improving the 
HTA are hypothesized below. 

2. Consider the analyst's task expertise level. As Dehoney (1995) 
suggested, the analyst needs to learn enough about the task to be able to ask 
the right questions and understand the answers before he or she can further 
explore the expert's heuristic process. During interviews, the expert sometimes 
forgot to explain important things or was unable to verbalize them, and having a 
certain level of domain expertise would have helped the analyst to be aware of 
any missing knowledge from the expert 's explanations. Besides that, the 
analyst's early lack of confidence in the task hindered building rapport with the 
expert throughout the analysis process. Thus, it is recommended that the SCM- 
HTA provide the following additional guidelines for analysts with limited task 
expertise: 

1 . During the preparation, review basic reference materials and try to 
become familiar with key concepts and jargon in the field. 

2. It is likely that the more complicated the version of the task is, the more 
task knowledge is required for the analyst. Thus, the analyst with 
limited task expertise should begin by identifying the simplest version 
of the task, rather than trying to expand the analysis to the next 
version. 

3. Develop strategies for working with multiple experts. Having multiple 
experts was beneficial, but working with more than one expert required special 
strategies for the analyst. It is recommended that the analyst start working with 
the least experienced expert, finish the first round with him or her, and then start 
the next round with the next least experienced expert. The less experienced 
experts' knowledge seemed to be less automatized, and thus those experts were 
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more conscious of their own heuristic knowledge. They tended to provide more 
information than the more experienced experts. Later, the analyst should ask the 
more experienced expert to review the results from the previous interviews. The 
more experienced experts provided more insightful information, which made 
them better reviewers. Nevertheless, it worked well to have a focus-group 
interview with multiple experts in the beginning, to reach consensus on the 
simplest version of task. 

4. Provide reference material during the interview. When asked to describe 
the task performance process relying on his or her memory, each expert tended 
to be rather abstract in his or her explanations and forget to tell the details. 
Having some kind of reference material (e.g., index cards summarizing critical 
incidents during the task performance or video/audio tapes of the task 
performance) provided contextual information when needed and thus helped the 
expert to be more precise. During the iterative interview process, the visual aid 
also helped the expert to keep on track. Especially under circumstances with 
time constraints, using index cards was a way to save time. 

Study 2 - The HTA Method Applied to Tutoring on Writing Skills 

Methods 

As in Study 1 , the purpose of this study was to improve Reigeluth’s HTA process 
by using the formative research methodology. This study followed the steps 
suggested by Reigeluth and Frick outlined in study 1. 

Task 

The heuristic task chosen for this study was tutoring university undergraduate 
students who needed extra assistance with their writing skills. Specifically, the 
task concerned the decision-making process in which an expert writing tutor 
engages to determine the direction and focus of the tutoring session. 

As Reigeluth (1999) pointed out, tasks can be based on a combination of both 
procedural and heuristic knowledge. Since the goal of this study was to focus on 
heuristic elements in this analysis, the task was selected to meet the following 
criteria: 

1. The task must involve higher-level writing concerns such as unity, coherence, 
and voice. 

2. The task must occur during a one-on-one, one-hour-long session where the 
expert has the full attention of the tutee, rather than as part of a classroom 
exercise 

3. Any procedural elements will primarily occur during the introduction and 
conclusion phases of the tutoring session. 
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By its very nature, a tutoring session requires a lot of heuristic expertise, 
because it is determined more by events that occur during the tutoring event than 
by any predetermined procedural steps. What occurs during the tutoring session 
depends on both the writing situation and the tutee. The writing situation 
includes why the tutee is being tutored, the relationship between the tutee and 
the teacher, the interest level of the topic being written about, and the number of 
drafts already written. The tutee includes any previous experiences, both 
positive and negative, that the tutee brings to the tutoring session. 

Participants 

Two experts were chosen based on their level of expertise and the approval of 
their supervisors. The first expert was completing his Masters in English and 
was in his second year of tutoring. The second expert had tutored during the last 
two years of her BA in English and was entering an MA program in English. Both 
experts had extensive experience tutoring all levels of writing students, and both 
were highly recommended by their writing center supervisors. 



Insert Table 5 about here 



A third tutor was also recommended and interviewed as a potential participant in 
this study. However, the recommendation came without the experience and 
evaluation credentials listed above, so he was not included in this study. 

Data Collection Methods 

Interview: As in study 1, the main data gathering method was the personal 
interview, and the researcher both elicited heuristic knowledge (analyst) and 
conducted formative research (researcher). Two interviews were conducted 
within one week of the tutoring session that was being analyzed. Because of the 
tutors’ lack of time to spend on this research, each interview was limited to 
approximately 60 minutes. 

The researcher served as an analyst who used the HTA method with the task 
experts in order to determine the strengths and weaknesses of the HTA method. 
Both of the interviews were conducted within one week of the tutoring session 
that was being analyzed. Because of the tutors’ lack of time to spend on this 
research, each interview was limited to approximately 60 minutes. Prior to the 
interviews, each of the tutors was sent emails describing terms used, definitions, 
an outline of the interview questions, and a brief explanation, written in their 
terms, of the purpose, expected results, and use of this research. Before the 
actual interview, the researcher reminded each tutor of what was sent to them 
earlier and asked if any terms or points needed to be clarified. At this time, the 
researcher also pointed out to each expert writing tutor that a) it was unclear 
whether the task about to be analyzed was actually based on heuristic 
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knowledge and b) it was unclear whether the questions would be able to access 
that knowledge. This was done to reduce any anxiety that the writing tutor might 
experience if she or he could not produce information that the researcher 
desired. 

Because of the experiences of researcher #1 and the recommendations of 
researcher #3, this study used index cards (for the expert to read and hold) for 
recording each question, recording the expert’s response, making the interview 
process and results more open and visible (rather than hidden away on the 
researcher's legal pad), and placing the focus on the cards (knowledge 
generated) rather than on the researcher. Both of the expert writing tutors 
referred to the cards during the interview and made changes to their comments 
as the interview progressed. 

Because the writing tutors did not have the time to be repeatedly interviewed as 
in Study 1 , this study did not attempt to identify the simplest version of the task, 
nor did it verify the task with other expert writing tutors. For the first interview, 
the writing tutor provided an audiotape of the session to be analyzed, and the 
analyst, after previewing it, chose one decision point as the focal point for the 
interview. The researcher’s decision to choose the specific area to focus on 
does not reflect the HTA methods, but was made so that the majority of the 60 
minutes allowed by the expert tutor for the interview would be spent on eliciting 
heuristic knowledge. Other than this change, the first interview followed the HTA 
method as described earlier in that it asked the tutor to identify goals, 
considerations, causal factors, guidelines and explanations. 

Based on this first interview, the HTA methodology was altered slightly for the 
second interview so as to assist the tutor to better recall the tutoring situation. 

The analyst had the tutor respond to specific questions about the actual tutoring 
experience, the tutee’s characteristics, and the tutee’s essay prior to having the 
tutor recall and reflect on her decision-making processes. He then had the tutor 
identify the decision areas to focus on during the tutoring session. Afterwards, 
he had the tutor choose the concern that was most available to her. This, then, 
became the subject of the heuristic task analysis. 

Data Analysis and Interpretation Methods 

There were no follow-up interviews or member checks as in Study 1 to determine 
the validity of the tutors’ responses due to the writing tutors’ lack of time to spend 
on these tasks. However, because of the researcher’s expertise in this area, he 
informally concluded that the data collected was not spurious. 

At the conclusion of each interview, the experts were asked to review and modify 
what was recorded during the interview. The researcher asked each expert for 
ways to improve the interview process and to comment on its effectiveness in 
eliciting the knowledge underlying their decision-making thought process. Both 
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offered suggestions about ways to help them recall the previous tutoring situation 
and about the limitation of focusing on only one aspect of the tutoring process. 
The second expert confirmed what the first had concluded. After each interview, 
tentative changes were made to the HTA process. 

Results and Discussion 

As mentioned in Study 1, the formative research methodology is an iterative 
process whereby formative evaluation of one instance (application of the HTA 
method) is used to improve the HTA method, which is then used to carry out 
another instance. Altogether, the researcher created two instances of the HTA 
method. After each instantiation, the researcher collected strengths and 
weaknesses with the HTA method as applied to writing tutors and implemented 
refinements to be used during the next instantiation. 

In the interview after the first instantiation of the HTA method, the first tutor 
mentioned how the process helped him think about his own tutoring strategies. 
He also mentioned that having to recall from memory a tutoring session that was 
done even within the last seven days was not easy. The tutor suggested the 
following refinements to the HTA method. 1) The top-down process seemed 
effective. 2) Because the tutor experienced some difficulty recalling the specific 
tutoring session, the analyst (researcher) asked some specific questions about 
the tutee, the paper, and the tutee’s reactions to the tutor’s suggestions. Both 
tutors said this helped them get into the flow of the previous tutoring session, and 
the researcher observed a marked increase in awareness and confidence after 
assisting the tutor’s recall. 3) 5x8 cards were effective in that the tutor referred 
back to them to align his insights into the tutoring process with previous 
statements. 4) The tutor, when identifying the guidelines, focused more on how 
to hold an effective tutoring session than on what influenced his decision to focus 
on a specific tutoring objective. In addition, 5) the researcher suspected that the 
results of the HTA might have been richer if the task expert (tutor) had been 
given more control over the decision point selection process. 

Considering these findings, changes were made to the HTA method, which was 
then used to carry out the second instantiation. In the post-analysis interview, the 
second task expert mentioned that, after discussing the context of the tutoring 
session, she had a clear picture of what was going to be analyzed. The 
researcher also noted the relative ease with which this expert could talk about 
the decision point as compared to the previous writing tutor. The researcher 
believes that helping the expert recall the task, the context of the tutoring 
session, the tutee, and the tutoring session caused the increased ability to 
discuss the area. Therefore, the findings from the second instantiation include 
the following. 1) Helping the tutor to access a recent tutoring session created a 
rich context to draw from. 2) Helping the tutor access the context of the tutoring 
situation, though, did not prevent the tutor from referring to the general goals 
when listing guidelines rather than the specific decision point of the task. 3) 
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Having the tutor list and choose the decisions she made during the tutoring 
session was time consuming but resulted in richer heuristics. 

In summary, an important concern involves the task expert’s tendency, when 
explicating the guidelines, to focus on the goals and not on the decision points for 
attaining the goals. During both instantiations, the writing tutors gave the 
guidelines they used for deciding on the goals of the tutoring situation rather than 
giving guidelines for deciding how to attain a goal during the tutoring session. 
When this occurred, the researcher gently prodded the experts to focus on the 
decision points rather than the goals. However, when the experts could not 
provide that information, the researcher decided to review previous sections and 
then ask that question again. After the experts referred to the goals again, the 
researcher decided not to push the expert any further, seeing that they both were 
unable to provide that information. 

Another concern involves the first expert’s difficulty in recalling the tutoring 
session despite the fact that the session occurred only one week prior to the 
interview. Measures taken to assist the second expert’s ability to recall the 
tutoring session showed a dramatic improvement. Table 6 presents a chart 
outlining what worked and what needed improvement for this instantiation of the 
HTA method as applied to tutoring. 



Insert Table 6 about here 



Study 3 - Selecting Artwork for a Commercial Product Line 
Methods 

The third study tested the HTA method in a corporate setting. As in the previous 
two studies, formative research was the methodology, using a designed case to 
generate possible improvements in the HTA method. 

Corporate executives want a “big bang for their buck,” and analysis is often 
looked upon as a time-consuming activity with questionable impact. The aim of 
this study was to develop a rapid, high-impact version of the HTA method for 
corporate settings. Thus, the study was designed to provide insight into the 
following research questions: 

1 . How can the speed and effectiveness of the HTA method be improved 
for eliciting, analyzing, and representing heuristic knowledge from 
experts in corporate settings? 

2. What guidance could be added to the method to assist analysts in 
corporate settings? 




24 



Heuristic Task Analysis 



The time constraints for this study dictated that the research be limited to a single 
interview cycle with one task expert, lasting no more than a total of three to four 
hours. 

Task 

The heuristic task chosen for this study was deciding whether a submission of 
artwork was suitable for one of the company’s product lines. This is a judgmental 
decision-making task requiring a fair amount of experience and know-how. The 
task expert verified that the task was important to the company, that she was 
considered to be an expert at the task, and that it was not easy to articulate the 
expertise required to perform the task. The heuristic nature of this task was 
verified by an expert in the HTA method, Charles Reigeluth. 

Participant 

The task expert was recruited by calling a local business that had collaborated 
with the Instructional Systems Technology Department at Indiana University in 
the past. A manager in the design department enthusiastically agreed to 
participate in the study. The expert and the analyst discussed possible complex 
decision-making tasks over the telephone and came to an agreement on an 
appropriate task. 

Data Collection, Analysis, and Interpretation Methods 

Interview: The researcher/analyst conducted two audio taped, one-and-a-half- 
hour interviews with the participant (task expert) in a conference room at the 
expert’s place of business. The analyst/researcher referred to an interview sheet 
(described below) to ensure that he was adhering to the guidelines of the HTA 
method, although he also allowed the interview to be somewhat unstructured as 
seemed appropriate to gather the heuristic knowledge and data for improving the 
HTA process. After one-and-a-half hours he reached a saturation point in terms 
of gathering the essence of the task and the key heuristics and concluded he 
could not effectively continue the analysis without first going back to his office 
and organizing the information collected. The expert agreed to continue the 
interview the following week. 

The analyst/researcher logged “significant chunks” of the audio tape on 3” x 5” 
cards. His criterion for “significant chunks” was any piece of knowledge that fit 
into one or more of the types of knowledge listed in the HTA method. He 
examined these knowledge elements to determine what missing ones he 
needed to ask about in the follow-up interview. Then in the role of researcher, 
he re-examined the interview results for deviations from the HTA method to see 
where the method was effective in eliciting heuristic expertise and where 
deviations were helpful. He discussed his preliminary findings with Reigeluth 
and worked with him to plan the second interview. 
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The analyst/researcher conducted a second, audio taped, hour-and-a-half-long 
interview to fill in gaps in the analysis and verify with the expert that the heuristics 
corresponded to the expert’s understanding of the task. After the second 
interview, the analyst/researcher again analyzed the audio tape to fill in gaps in 
the heuristic knowledge generated. Then in the role of researcher he analyzed 
the interview process as recorded in the audio tapes to determine how the 
process deviated from the HTA guidelines and to develop recommendations for 
possible improvements in the HTA method. 

In an attempt to ensure that the interview conformed to the HTA method as much 
as possible, an interview sheet was developed and used (see Appendix B). The 
steps on the interview sheet were written as questions to be asked using 
terminology that seemed understandable to a layperson. The rationale for this 
was that one of the goals of the study was to test the HTA method in a business 
context that placed a premium on short cycle time (e.g., short interviews). Using 
lay terminology was hypothesized to decrease the net amount of time for 
analysis by eliminating time for explaining academic terms. The interview sheet 
was edited by Reigeluth to verify that it accurately represented the HTA method 
in its current form. 

It is important to note that the HTA method is a set of guidelines to be used 
heuristically, not as a step-by-step checklist of questions to ask. One limitation of 
this study was this researcher’s limited experience as an “HTA analyst,” who 
would ideally have the guidelines in mind and would have the know-how to 
choose the appropriate guidelines most of the time and to adeptly recover from 
inappropriate choices the rest of the time. 

Results and Discussion 

The research questions for this study addressed the speed and effectiveness of 
the HTA method for analysts in corporate settings. 

Speed of the HTA method. The task expert originally agreed to three hours of 
interview time, and the analyst/researcher complied with this limit. Afterwards the 
expert said the amount of time spent was within acceptable limits. The 
analyst/researcher also spent about six hours analyzing the interview audio tapes 
to synthesize the expert’s task knowledge. 

Approximately half of the interview time was spent eliciting and categorizing the 
expert’s heuristic task knowledge, while the rest of the time was spent developing 
rapport, explaining the process, selecting and describing the task, and discussing 
associated tasks. Analysis of the discussions on associated tasks revealed that 
at some points these discussions helped the analyst/researcher better 
understand the context of the main task, while at other points these discussions 
were cases of getting sidetracked from the purpose of the analysis. 
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The time spent on formative research on the HTA method is not included in the 
above six hours of analysis, as this has no bearing on the amount of time spent 
analyzing the heuristic task knowledge. 

The analyst/researcher found a number of areas in which the speed of the HTA 
method might be enhanced. Two are discussed in this section. The others are 
the result of improving the effectiveness of the method and are discussed in the 
next section. The analyst/researcher noticed that almost an hour was spent 
identifying the simplest version of the task and distinguishing it from other 
possible versions. This can be important for training purposes, as outlined in the 
SCM methodology. However, in a business context, there can be other purposes 
for conducting the heuristic task analysis. The results of a heuristic task analysis 
can be used to generate job aids for experts, to help designers structure 
knowledge-management systems, and for other purposes. If training is not the 
primary purpose, then the analyst might choose to spend less time identifying the 
simplest version and other versions (Step 2. Identify the first learning episode). 
Such information might still be useful for distinguishing experts and novices, even 
though sequencing course material is not of concern. In this study, the 
analyst/researcher concluded that this step could have been concluded with 
significantly less time (approximately 20 minutes less), without diminishing the 
quality of the results. 

Recommendation: Unless using the HTA method specifically for training 
purposes, perform Step 2, "Identify the simplest version," only if needed to 
distinguish between experts and novices or as one way of helping the expert 
access tacit knowledge. As the expert examines various instances of a task in 
search of heuristics, it may be helpful to distinguish between simpler and more 
complicated versions. 

Analysis of the data showed that almost an hour was spent on tasks closely 
related to the task studied. Part of this time was spent on placing the main task in 
the context of other tasks and part on analyzing the related tasks. When the 
expert explained the nature of the task in general (Step 1 .3), it was helpful to 
have the expert also mention tasks that are closely related. This seemed to 
speed up the overall process by avoiding confusions and time wastage caused 
by mixing details of two different tasks at a later point in the interview. Tasks can 
be related in various ways. They might occur in tandem with, or come right 
before or after, the chosen task, or they might have a causal relationship with the 
chosen task. The analyst needs to be aware of these tasks and be alert for 
digressions. 

Recommendation: When performing Step 1.3, have the expert briefly discuss 
any closely related tasks and clearly distinguish between the main task and the 
related tasks during the remainder of the task analysis. 
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Increasing the effectiveness can also increase the speed of the HTA method. 

This is addressed in the next section. 

Effectiveness of the HTA method. The HTA method seemed to be effective in 
its primary function of eliciting heuristic knowledge from the expert. At one point, 
the expert apologized for not being able to explain how she knew the artwork 
was right for their products, but then she proceeded to discern several criteria 
that were fundamental to making such a decision. The expert was satisfied with 
the criteria and indicated that they explained how she made the decision on 
selecting artwork, although she did not consciously use the criteria as such. 

The types of knowledge delineated in Section 3.2 of the HTA Method were found 
to adequately cover the range and types of task knowledge described by the 
expert. The analyst/researcher did, however, have problems managing the two 
tasks of classifying the expert’s knowledge and directing the interview to dig 
deeper into the expert’s tacit knowledge. More practice with the methodology 
should alleviate this. The analyst/researcher noticed during the analysis that 
certain verbs used by the expert were indicators of tacit knowledge. Examples of 
these verbs are: know , like, feel, see, determine, understand, and decide. When 
the analyst/researcher asked the expert why she liked a certain piece of art, she 
struggled at first to find reasons, but eventually she isolated specific 
characteristics that distinguished artwork she liked from pieces that she did not 
find acceptable. 

Recommendation: Step 3.2 should include guidance specifically focusing on the 
expert’s communication. This analyst/researcher recommends adding this 
guidance: The expert’s language will often contain clues that can help the analyst 
find tacit knowledge. Especially common are words, such as know, like, feel, see, 
determine, understand, and decide. The analyst should probe the expert for 
explicit details that can be communicated to others by asking, for example, what 
they know, like or feel, what criteria they use to determine something, and how 
they decide. 

Upon further analysis and reflection the analyst/researcher developed a model 
for thinking about how to view the knowledge elicitation and representation 
process, involving three components. The three components are: (1) the expert’s 
internal representation of her or his knowledge — the tacit knowledge, (2) the 
manner in which the expert expresses the internal representation using language 
or other means and the manner in which the analyst guides the interview or 
elicits the tacit knowledge — the expert/analyst dialog, and (3) the manner in 
which the analyst and the expert decide to represent the knowledge — the explicit 
knowledge. 

This model can be useful to the analyst for eliciting tacit knowledge and for 
transforming the tacit knowledge into explicit knowledge. For example, two 
experts might use the same verbs, such as know and like, to communicate 
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different tacit knowledge, or they might use different words to communicate what 
is basically the same tacit knowledge. Furthermore, different people might prefer 
one explicit representation to another, as we can see with the oft-discussed 
differences between visual, oral, and kinesthetic learners. This three-component 
model might be useful in helping the analyst to conduct the HTA process. 

The guidance in "Step 3, Analyze the organizing content" appears to focus on 
how to elicit and how to represent the tacit knowledge. The elicitation guidance 
appears to reflect a single elicitation style, focusing on a single specific 
performance at a time. For example, Step 3.1 has the expert focus on one 
specific performance of the task, whereas the expert in this study sometimes 
hopped from one instance to another as she attempted to articulate her tacit 
knowledge. Further research could focus on whether the elicitation guidance 
provided by the HTA method should be expanded to provide a variety of types of 
elicitation styles and techniques for varying situations, experts, and analysts. 

Recommendation: Guidance in Step 3 should clearly distinguish between 
guidelines for eliciting and those for representing the tacit knowledge. 
Furthermore, elicitation guidance should be generalized and expanded. 
Specifically, under Step 3.1 , this analyst/researcher recommends adding the 
following bullet point: Sometimes task experts find it helpful to compare two or 
more instances when trying to articulate their tacit knowledge. 

The guidance focusing on how to represent the tacit knowledge does not attempt 
to distinguish between representations of tacit and explicit knowledge. The theory 
of knowledge used in the HTA method breaks the knowledge into task goals and 
decisions with several types of knowledge that the expert uses in making 
decisions and carrying out the task. In the HTA method these types of knowledge 
are meant to represent the expert’s tacit knowledge. The analyst is left to assume 
that this explicit knowledge framework is a good fit for the tacit knowledge of the 
expert, but further research on representing expert tacit knowledge may show 
that the choice of explicit representation depends on the expert’s tacit 
knowledge. Clearly, the ultimate goal of the HTA method is to provide a useful 
explicit representation of the tacit knowledge. 

Recommendation: Additional guidance should be developed on how to 
represent explicit knowledge. Although this analyst/researcher has only begun to 
research this point, such guidance could come from fields such as task analysis 
or the expert’s specific field. One area of interesting research would be 
collaboration between the expert and the analyst to develop an explicit 
representation for knowledge deemed critical. 

Guidelines for analysts. Throughout the two interviews, the analyst/researcher 
made a conscious effort to avoid academic jargon, and the expert seemed to 
rapidly understand everything the researcher was saying. In moments where the 
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analyst/researcher caught himself using a technical term, he laughed it off with 
the expert and used the moment to increase rapport. 

The analyst/researcher wrote down many of the points the expert made on index 
cards. It was helpful to have these out on the table, and both referred to them a 
number of times during the two interviews. The cards also helped the 
analyst/researcher stay on track or get back on track, when sidetracked. 

Table 8 presents a summary of the strengths and weakness of the HTA Method 
in Study 3. 



Insert Table 8 about here 



CONCLUSIONS AND RECOMMENDATIONS 

Similarities and Differences among the Results 

The following summary makes it clear that the three studies have some common 

findings, but that each also has unique findings. 

Study 1 

• In interviewing several experts simultaneously, it helped to have them all think 
of the same case during the analysis process. Prior to doing that, differences 
in heuristic knowledge were confusing. However, based on this experience, it 
may be better to complete the analysis with one expert, then conduct the 
analysis with another, and so forth, so that you create a separate analysis for 
each expert rather than one single analysis. The advantages appear to be 
that (1) each expert knows what has transpired to date in their analysis and 
(2) similarities and differences in heuristic knowledge of the experts are 
catalogued in separate analyses, making subsequent comparisons and 
discussions easy. 

• When working with more than one expert (which is highly recommended), it 
worked better to start with the least experienced one, finish with him/her, and 
proceed to progressively more experienced ones, one at a time. However, it 
appears that it would have been beneficial to have a focus group interview 
with all the experts in the beginning to reach consensus on the simplest 
version of the task. 

• It would have been better if the analyst had started with at least a basic level 
of expertise in the heuristic task, so that the analyst could more easily 
understand the terms and ideas that the experts used and could prompt the 
experts when they were having difficulty expressing their heuristic knowledge. 
Such expertise could also improve the analyst's rapport with the experts. 
Furthermore, it would have been helpful for the analyst to observe the 
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performance of the chosen case. 

• It was helpful to summarize the key incidents from the chosen case on index 
cards and refer to them during the interviews. The benefits were: (1) it helped 
the experts to recall details about the task performance process (2) it helped 
both analyst and experts to see the flow of the task performance process and 
get back on track when the experts got off-task, and (3) it saved a lot of time 
in revisiting previous points. 

• It worked better to have the experts identify "freeform" whatever knowledge 
they could about the task and later try to categorize it as to types (steps, 
guidelines, explanatory models, descriptive models, and 
metacognitive/decision rules) and fill in any gaps, rather than trying to get the 
experts to identify knowledge systematically within each type of knowledge. 

• Categorizing the expert's knowledge as steps, guidelines, explanatory 
models, descriptive models, and metacognitive/decision rules was difficult, 
and its benefits may not have outweighed its costs. The benefits need to be 
clarified for the analyst. 

Study 2 

• Identifying knowledge within categories seemed to work well. (Note that this 
contradicts a finding from Study 1.) 

• It was beneficial to help the expert reconstruct and articulate a specific case 
for which she performed the desired version of the task and within which her 
decisions (and goals) were formulated. 

• It was beneficial to periodically ask the expert some questions about the 
chosen case, to keep the analysis focused on the flow of that version of the 
task. 

• It was useful to help the expert think about ways the specific case fell short of 
how it should have been done (an ideal case for this version of the task) and 
to have the expert offer guidelines for how this specific case should have 
been done. 

• It helped to summarize on index cards the key incidents from the chosen case 
and refer to them during the interviews, to help the expert align her insights 
into the task with previous statements. 

• It was helpful for the analyst to have the expert explain the goals in task- 
specific terms rather than in abstract terms. 

Study 3 

• When performing Step 1 .3, it was helpful to have the expert briefly discuss 
closely related tasks and clearly distinguish between the main task and the 
related tasks during the remainder of the analysis. 

• Because training was not a primary purpose of the task analysis, performing 
Step 2 seemed most helpful in distinguishing between experts and novices 
and assisting the expert in articulating the tacit knowledge. More time would 
be spent on this step if the analysis were being done for training purposes. 
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• It would have been helpful to have guidance in Step 3.2 about using particular 
words (such as know, feel, like, see, understand, determine, and decide) as 
the focus for probing the expert's tacit knowledge. 

• Additional guidance on elicitation strategies might be helpful. One elicitation 
strategy might focus on types of instances, for example, comparing similar, 
different, or easy and difficult instances. Another elicitation strategy might 
focus on how the task expert approaches the instances, for example, 
comparing, contrasting, narrating, or thinking of metaphors for instances. 

Suggestions for Improving the HTA Process 

The following is a tentative revision of the HTA process based on the findings of 

these three formative research studies. The changes and additions are in italics. 

Phase I. Prepare for Analysis and Design 

1. Decide on a task to analyze and be clear about the reasons for analyzing it. 

2. Make sure you have enough task knowledge to have a good command of terminology 
and key ideas. 

• If you don't have enough task knowledge, review basic reference materials and try 
to become familiar with key concepts and jargon in the field. 

• If you have limited task knowledge, it would be better to begin by identifying the 
simplest version of the task, rather than trying to expand the analysis to the next 

version. 

3. Make sure you have enough knowledge about the uses of the task description. 

• If the task description will be used primarily for deciding on the content and 
sequence of instruction, identify the characteristics of the learners in general and 
identify the delivery constraints of the instruction in general. 

4. Arrange to interview multiple experts. 

• Identify at least 2 or 3 experts to interview. 

• Plan to complete the analysis with one expert before initiating the analysis with 

another. 

• Plan to interview the least experienced expert first and proceed to interview 
progressively more experienced experts in order. The less experienced experts are 
likely to have less automatized knowledge, and thus be more conscious of their 
own heuristic knowledge. They tend to provide more information than the more 
experienced experts. On the other hand, the more experienced experts can 
provide more insightful information, which makes them better reviewers. 

• Ask one or more of the task experts to record their performance of a very simple 
version of the task, and review the recorded material in advance of the analysis; or 
observe the task expert's task performance (more than once if possible). 

5. Prepare in conjunction with the first (next) task expert. 

• Establish rapport with the task expert. 

• Introduce the HTA method to the expert. 

• Explain basic terms (i.e. guidelines, explanatory models, etc.). 

6. Prepare for the interview. 

• Prepare interview materials (i.e., index cards to summarize cn'tical incidents during 
task performance). 

• Practice the HTA interview process if you are not very experienced in it. 
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• Arrange the interview logistics (e.g., reserve a conference room where you can 
work without interruptions). 

Phase II. Identify the First Learning Episode 

7. Identify the simplest version. Hold a focus group interview with multiple task experts, 
and help them to reach consensus on the simplest version of the task that is fairly 
representative of the task as a whole. Also help them to describe the conditions that 
distinguish that version from all other versions. 

• You may want to use some other criteria in addition to simple and representative, 
such as common (how frequently performed the version of the task is) and safe 
(how much risk there is to the learner and/or the equipment). 

• It may be helpful to have the expert briefly discuss closely related tasks and clearly 
distinguish between the main task and the related tasks during the remainder of 
the analysis 

• Ask the task experts to recall the simplest case they have ever seen. The simplest 
version will be a class of similar cases. Then check to see how representative it is 
of the task as a whole. 

• It may be helpful to start by identifying some of the major versions of the task and 
the conditions that distinguish when one version is appropriate versus another. 

• Thinking of different conditions helps to identify versions, and thinking of different 
versions helps to identify conditions. Hence, it is wise to do both simultaneously 
(or alternately). 

• There is no single right version to choose as the "simplest." It is usually a matter of 
trade-offs. The very simplest version of the task is usually not very representative 
of the task as a whole. The more representative the simple version can be, the 
better, for it provides a more useful schema to which learners can relate 
subsequent versions. 

• It is wise to go through this process with several task experts together and reach 
consensus before going on to Step 8. You may find it necessary to take steps to 
resolve differences of opinion about which is the best “simplest version” to use. 

8. Analyze the organizing content. With the least experienced expert you have not yet 
interviewed, analyze the organizing content (mostly heuristics and descriptive theories) 
for this version of the task. 

8.1 Review the recorded material (or any other visual aid) with the task expert. 

8.2 Ask the task expert to think of and describe one specific performance of the 
selected version of the task to focus on for your analysis, or ask if a videotaped 
performance would be a good case for you to focus on with the expert during 
the analysis. 

• It is often helpful to have a videotape of a typical performance of the 
simplest version of the task, so you and the task expert can review it 
during the analysis process, but asking the task expert to recall one 
specific performance and keep it in mind throughout the process is a 
more convenient and inexpensive, albeit often less effective, alternative. 

• If you don't have a videotape, It may be helpful to have the expert 
describe contextual information and particulars of the specific 
performance, describing how the expert began the case, how it 
progressed (in sequence), how participants reacted, and how the expert 
dealt with any problems that arose. 

• It may be helpful to prioritize the problems/concerns that arose and 
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the decisions/actions that the expert used to deal with them. 

8.3 Decide whether to use a top-down or bottom-up approach to analyzing the 
content (the knowledge upon which the expert's performance is based). If top- 
down, use Step 8.4 and skip Step 8.5. If bottom-up, skip Step 8.4 and use 
Step 8.5. 

8.4 If top-down approach, start by identifying the general categories of knowledge 
that an expert uses, then proceed to analyze each. 

• Ask the task expert: 

a) to describe each decision that the task expert made, 

b) to identify the kinds of knowledge that the task expert drew upon 
to make the decision, and 

c) to describe the specific knowledge within each kind of knowledge 
that the task expert used. 

• The kinds of knowledge are likely to include: 

- steps (procedural knowledge), 

- guidelines or rules of thumb (heuristic knowledge), 

- explanatory models (which explain why the guidelines work), 

- descriptive models (which describe the phenomena with which the 
task expert interacts), and 

- metacognitive/decision rules (which the task expert uses to decide 
which steps, guidelines, and descriptive knowledge, to use when). 

• It is generally helpful to start by asking the task expert if there are 
any steps or phases of activities that are always performed for this 
version of the task. If so, perform a procedural task analysis to identify 
the sequence of steps and to see if any of those steps can be broken 
down into substeps, but those substeps must be ones that an expert 
thinks of and uses routinely in performing that version of the task. 

• For guidelines, use the following process (See the example in 
Figure 1 below.): 

1 . Identify the goals for this specific performance of the task under 
its conditions. It may help to have the expert explain the goals in 
task-specific terms rather than in abstract terms and to think of 
the goals as ideal outcomes. 

2. Identify all the important considerations for attaining each goal. 
Considerations are the major categories of causal factors that 
influence performance of the task. If there are a lot of causal 
factors for a consideration, it is useful to identify 
subconsiderations for it. 

3. Identify all the important causal factors that relate to each 
consideration (or subconsideration). 

4. Analyze each causal factor to identify all guidelines (prescriptive 

principles or “rules of thumb”) that an expert uses to account for 
this consideration. 



Insert Figure _ about here 
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• For explanatory models, use the following guidelines: 

For each guideline, ask the task expert for the reasons why 
s/he believes it works. 

For interrelated guidelines, you are likely to identify a set of 
related reasons that constitute a causal model or models. Be sure 
to look for multiple causes for each effect and multiple effects for 
each cause. Also look for chains of causes and effects, and 
explore probabilities for each causal factor to have each effect. 

• For descriptive models, use the following guideline: 

Ask the task expert what phenomena influenced this 
particular performance of the task. Try to identify all causal 
relationships that characterized those phenomena. 

Be sure to look for multiple causes for each effect and 
multiple effects for each cause. Also look for chains of causes 
and effects, and explore probabilities for each causal factor to 
have each effect. 

• For metacognitive/decision rules, use the following guideline: 

Find out what rules the task expert used to decide when to 
use which steps, guidelines, and descriptive models during the 
specific performance of the task being analyzed. 

• It is wise to query the task expert about any of these kinds of 
knowledge that are not initially described to you for each decision the task 
expert made in this specific performance of the task. 

• If the expert uses words such as know, feel, see, understand, like, 
determine, and decide, that may be an indication that heuristic knowledge 
underlies that particular performance. 

• It is often helpful to periodically ask the expert some questions about 
the chosen case, to keep the analysis focused on the flow of that version 
of the task. 

• It is useful to help the expert think about ways the specific case fell 
short of how it should have been done (an ideal case for this version of 
the task) and to have the expert offer guidelines for how this specific case 
should have been done. 

• It is wise to have some kind of reference material (e.g., index cards 
summarizing critical incidents of the task or videotapes of the task) to 
provide contextual information and cues and to help the expert be more 
precise. During the iterative interview process, the visual aid also helps 
the expert keep on track. Especially under circumstances with time 
constraints, using index cards is one of the ways to save time. 

• It may be helpful to use index cards for all of these kinds of 
knowledge, filling them out with the task expert during the analysis 
process with one piece of knowledge per card, and arrange the cards in 
some order on a table in front of both of you, so you can easily switch 
from one part or aspect of the task to another. 

8.5 If bottom-up approach, ask the expert to describe each decision that s/he made 
and the process through which s/he went to make each decision. 

• After the interview, try to categorize each piece of heuristic 
knowledge according to these categories: 
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- steps (procedural knowledge), 

- guidelines or rules of thumb (heuristic knowledge), 

- explanatory models (which explain why the guidelines work), 

- descriptive models (which describe the phenomena with which the 
task expert interacts), and 

- metacognitive/decision rules (which the task expert uses to decide 
which steps, guidelines, and descriptive knowledge, to use when). 

• Be sure to "member check" the interview results with the expert in a 
later interview to verify/identify the types of knowledge underlying each 

decision. 

8.6 Ask the task expert to think of similar performances of the task that are within 
the realm of the version of the task you are currently analyzing. Use each such 
performance to broaden the steps, guidelines, explanatory models, descriptive 
models, and metacognitive/decision rules so that they represent the knowledge 
bases the task expert uses to deal with all performances for that version of the 
task. 

8.7 Repeat this entire process (Steps 5 - 8.6) with the next least experienced task 
expert to identify any alternative views of the task and the knowledge that 
underlies its performance. 

• For each more experienced expert, you should summarize the 
previous description of the task and ask the expert to review it, in an 
effort to reconcile conflicts and select among alternative ways of thinking 
about and performing the task. The more experienced experts can 
provide more insightful information, which makes them better reviewers. 

Formative research data indicate that this revised HTA process would have been 
more effective for the three cases investigated here. It remains to be seen 
whether or not this revised process will also work well for analyzing other tasks 
that have heuristic elements. The data in this study indicate that much additional 
guidance is still needed for conducting a heuristic task analysis. It is our hope 
that this study will encourage others to conduct additional research to improve 
the available guidance for analyzing heuristic tasks. 
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Table 1 

Comparison of Various Knowledge Elicitation Techniques 



Technique 



Description 



Advantage 



Disadvantage 





• involves familiar 


• offers minimal 


• difficult to interpret 




tasks, simulated familiar 


interference with the 


the data 




cases, limited information 


expert's task and 


• potential influence 




tasks, constrained 


environment 


of the observer’s 


Observation 


processing tasks, and 


• helps identify 


presence on expert’s 


tough cases 


problem-solving strategies 


behavior 






that are not consciously 


• expert’s underlying 






accessible or to verify 


reasoning is usually not 






expert’s description of what 
he or she actually does 


revealed by their actions 



► goal-oriented — 
follows a predetermined 
format 



Structured 



Interview 



► provides more 
systematic and thus more 
complete coverage of the 
domain 

► improves efficiency 
and effectiveness of 
knowledge acquisition 

► can be applied to 

knowledge acquisition from 
multiple experts 



► requires more 
preparation time and 
domain knowledge on the 
part of the elicitor who 
determines the interview 
format 



Un- 

structured 



► free-form interviews 
in which neither the content 
nor the sequencing of the 
interview topics is 
predetermined 



> is able to elicit 
unanticipated knowledge 

► is best-suited for 
early knowledge elicitation 
sessions, for getting a 
broad view of the domain, 
or for establishing rapport 
with an expert 



► requires facilitating 
skills for the interviewer 

► experts tend to 
provide a reconstructed 
version of reasoning and 
omit some components 
that may be important to 
solving the problem 
because they assume 
them to be obvious 



• thinking aloud 


• no time delay 


• is most time- 


• protocol must be 


between the expert’s task 


consuming in terms of 


analyzed based on a 


performance and the report 


data analysis 


Protocol Analysis systematic breakdown of 


of the performance 


• subjective nature 


the information to produce 


• creates a detailed 


of data interpretation 


a structured model 


picture of the 


• gaps and jumps in 




representation 


verbalization 


• a way of organizing 




• is difficult to 


a mass of open-ended 




determine the 


material by objectively and 




appropriate categories 



Content Analysis 



systematically identifying 
specific characteristics 
► seeks regularities 
while doing hypothesis 
testing 



Discourse Analysis 



► a way of parsing an 
interaction between the 
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elicitor and the expert 



• a multidimensional 


• is effective in very 


• is restricted to 


scaling method to elicit 


complex applications 


analysis problems 


further conceptual 


• is useful in finding 


• is not likely to 


Repertory Grids knowledge about the 


patterns and structures 


provide much information 


domain 


• provides a 


about the procedural 




conceptual framework for 


knowledge 




knowledge 





Critical Incident 
Method/ 

Critical Decision 
Method 



► asks the experts to 
recall and retrospect about 
previously encountered 
cases and select cases on 
the basis of their 
importance guided by 
probe questions 



► captures the kinds of 
knowledge and experience 
involved in real-world 
decision making and 
problem solving 

► combines multiple 
basic techniques such as 
protocol analysis, case- 
based reasoning, 
structured interview, and 



» potential 

idiosyncracy 
» experts' self- 

structured recollections 
typically reveal what 
happened but offer little 
insight into why the 
judgments, assessments, 
and decisions that 
surrounded critical events 





• focuses on specific 


• many of the cases 




experiences, assuming that 


involve very old 




the remembering or 


memories, which 




enacting of a specific case 


increases the potential 




will cue some information 


for memory retrieval 


Case Study 


that may have been 


errors and 


Analysis 


otherwise overlooked or 


reconstructions on the 




forgotten 

• involves simulations 

in which the elicitor walks 
through the case with the 
expert step-by-step 


part of the expert 



Forward Scenario 
Simulation 



► makes use of 
simulation to focus on a 
case 



► the amount of 
knowledge required of 
the elicitor to describe the 
situation in explicit terms 
familiar to the expert 



O 
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Table 2 

Top-down Approach to Heuristic Task Analysis 



Method 


Task: Determine the media for a course 


1 . Identify the goals of the task (or 
subtask) 


* the media will help the learner to master the 
objective, 

* the media will be cost effective, 

* the media fall within the constraints for the course 
development and implementation. 


2. Identify the considerations for 
attaining each goal. (If there are 
lots of causal factors for a consid- 
eration, then it is helpful to also 
identify subcategories of 
considerations 


For the third goal above: 

• budget, 

• skills of personnel available to teach the course, 

• availability of equipment for the course. 


3. Identify specific causal factors 
for each consideration (or subcat- 
egory). 


For the third consideration above: 

• numbers of equipment, 

• scheduling of equipment, 

• alternative uses of equipment, 

• features (capabilities) of equipment. 


4. Analyze each causal factor to 
identify all guidelines an expert 
uses to perform this version of the 
task, that involve the causal factor. 


For all the above factors: 

• If an insufficient number of the equipment is available 
for the projected number of students, do not select 
that delivery system. 

• If the equipment is not available at all the necessary 
times, do not select that delivery system. 

• If the equipment is available and would otherwise go 
unutilized, there is a stronger need for you to select 
that delivery system. 

• If the capabilities of the equipment do not meet the 
instructional needs, do not select that delivery 
system. 

Note: these examples are illustrative, not exhaustive, 
and there may be more than one guideline for a 
causal factor. 


5. Identify any decision rules an 
expert uses to combine the guide- 
lines into a performance model. 


For you to do. 


6. Identify specific explanations as 
to why each of the guidelines works, 
and combine the explanations into 
explanatory models. 


For you to do. 


7. Identify a descriptive model for 
any objects involved in performing 
the task. 


There are no objects that one uses to select media for a 
course. 
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Table 3 

Participant Profile for Study 1 





Expert #1 


Expert #2 


Expert #3 


Status 


Professor in 
Counseling Dept. 


3rd Year 
Doctoral Student 


1st Year 

Doctoral Student 


Years of Experience 


more than 20 years 


5 years 


3 years 



Table 4 

Strengths and Weakness of the HTA Process in Study 1 



Strengths 



• In-person interview 
seemed to work well 
for eliciting the 
experts’ heuristic 
knowledge. 

• Working with multiple 

experts seemed to be 
beneficial. 

• In general, the HTA 
process seemed to 
be logical and 
sequenced 
appropriately to elicit 
experts’ heuristic 
knowledge. 



Weaknesses 



Problems 

• Experts were not always able 
to verbalize their heuristic 
knowledge during 
theinterview. 

• Simultaneously working with 

multiple experts was 
challenging in terms of 
checking and balancing. 

• Analyst’s lack of task expertise 

was disadvantageous in 
questioning and prompting 
the expert. 

• It was difficult to use the top- 
down approach during the 
interview. 

• Experts often got off-track or 

had difficulty in remembering 
details during the interview. 



Solutions 

> Incorporate various interview 
and observation techniques 
(e.g. t simulated videotapes, 
staged cases). 

> Offer strategies for working with 
multiple experts. 



> Acquire sufficient task expertise 
during the preparation. 



> Use bottom-up approach during 
interviews and categorize later. 

> Provide reference material 
during the interview. 



• There were many distractions 
when interviewing experts in 
their offices. 



> Reserve a conference room and 
check out the interview 
conditions. 
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Table 5 

Participant Profile for Study 2 





Expert #1 


Expert #2 


Status 


MA student 


Graduating BA student 


Years of Experience 


2 years as MA student 


2 years as BA student 


Evaluation 


Passed evaluation during each 


Passed evaluation during each 




term as a writing tutor 


term as a writing tutor 



Table 6 

Strengths and Weakness of the HTA Method in Study 2 



Strengths 


Weaknesses 


Problems 




Solutions 


* Top-down process 


• Expert may need 


> 


At the beginning of the 


seemed effective 


reassurance if he/she 




interview, reassure the 


* Interview process 


cannot respond to the 




expert that their task 


seemed effective 


questions designed to elicit 




expertise may not be 


* Researcher having 
sufficient domain 


heuristic knowledge. 




based on heuristic 
knowledge. 


knowledge was beneficial 


• Experts may have 


> 


Note when expert has 


• Use of 5x8 cards during 


difficulty choosing a 




difficulty and ask questions 


the interviews was 


decision point to focus 




to facilitate recall of the 


effective 


on. 




case. 


• Having expert list 


• Experts may have 


> 


Note when the expert has 


decision points and 


difficulty recalling or 




difficulty and ask questions 


choose one as the focus 


getting into the flow of the 




to facilitate recall of the 


of the interview seemed 
effective 


tutoring situation. 




case. 
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Table 7 

Strengths and Weakness of the HTA Method in Study 3 






Strengths 



Expert was satisfied 
with speed of HTA 
process 






Weaknesses 



Problems 

A lot of time was spent on 
identifying the simplest 
version. 



Solutions 

> Minimize step 2, simplest 
version, if analysis not done for 
developing training curriculum 



• HTA method was 

effective in eliciting • 

tacit heuristics 

• Interview sheet as 

modified for this 
study helped guide 
analyst and speed • 

up process 



A lot of time was spent on 
closely related tasks. 



Certain words, e.g., know, 
like , gloss over or 
camouflage tacit knowledge. 



In Step 1.3 clearly distinguish 
between main and related 
tasks, and keep distinction in 
mind throughout interview. 



Analyst should be alert for 
such words and probe for 
heuristic knowledge. 



HTA method reflects a single 
elicitation style. 



> Analyst should be aware of 
various ways to work with 
experts to elicit knowledge. 



HTA method reflects a single 
knowledge representation 
format. 



> Expert may best portray 
heuristics with diagrams, 
tables, etc., in addition to using 
words. 
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Appendix A 
Interview Questions 



Questions related to the nature of task in general: 

1 . How do you perceive group counseling? Is it procedural, heuristic, or 
both? 

2. If it's procedural, in what circumstances? 

3. If it's heuristic, in what circumstances? 

4. If it's both, in what circumstances? 

Questions related to the simplest version of the task: 

5. Could you think of any group counseling session, which is the simplest but 
still representative? 

• Tell me about the setting: 

- When did you lead the session? 

- How many participants did you have? 

- What kinds of people were they? 

- What was the goal of the session? 

- What are the main considerations to achieve the goal? 

• Tell me about major decisions you made during the session: 

- What was the procedure you follow (if there's any)? 

- What were the major decision-points at the session? (i.e., When 
did you intervene and how did you intervene?) 

• Tell me what made you to make such decisions: 

- What are the decision-rules at each decision-point? 

- What are the alternatives at each decision-point? 

- How does the decision rules work? 

- Could you explain why each of the rules worked or did not work? 

- How would you do differently if you encounter similar cases in 
the future? 
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Appendix B 

Interview sheet for the Heuristic Task Analysis Method 



Phase I - Prepare. 

1 . Prepare 

□ Nice office... 

□ The process we will be using... 

□ Describe the task to me in general. 

□ Describe the people who will be performing this task. 

□ What are the main costs and constraints in terms of teaching people 
how to do this task? 

Phase II - Identify the first learning episode. 

2. Identify the simplest version of the task and simplifying conditions. 

□ Recall the simplest case of you doing this task. Describe it very briefly. 

□ What conditions distinguishes this version of the task from more 
complex versions? 

□ Let’s identify some major versions and any conditions that distinguish 
them. 

□ Is this the simplest, most representative task? 

□ How representative of the task is it? Does this example occur 
frequently? Is it going to be safe to do in training? 

□ Is this a fairly complete list of the simplifying conditions? 

3. Analyze the organizing content (mostly heuristics and descriptive theories) 

□ Think of a specific time you performed this task (or videotape). 

□ What are some key decisions you made? 

□ What knowledge did you draw upon to make this decision? (What 
were you thinking when you made this decision?) 

□ Steps? (procedural knowledge) 

□ Descriptive models? (which describe the phenomena with which 
the SME works), 

□ When you performed the task, what things were you dealing 
with? 

□ Can you think of any sort of cause/effect relationships far those 
things, phenomena? 

□ Multiple causes, multiple effects? 

□ Chains of causes and effects? 

□ What are the probabilities for these things happening? 

□ Guidelines? (heuristic knowledge) 

□ What are the goals for this instance of the task under the 
conditions we talked about? 

□ What important considerations did you have in achieving the 
goal(s)? What are the main things you had to consider or pay 
attention to get achieve the goal(s). 
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□ What are the critical events and inputs you have to do or think 
about to deal with each consideration? 

□ What sort of guidelines or rules of thumb would help someone 
handle this consideration? 

□ Explanatory models? (why the guidelines work) 

□ Why do you think this guideline works? 

□ Are any of the guidelines interrelated? What do you think are 
the reasons that these guidelines work? 

□ What’s the probability of each of the critical events happening? 

□ Metacognitive/decision rules? (when to use which steps, 
guidelines, and descriptive knowledge). 

□ How did you decide when to use these guidelines/rules while 
you were working on this task? 

□ Recall a similar example of this task that constitutes the same type of 
version of the task. Go to beginning of Step 3. 

4. Analyze the supporting content. 

□ What additional information, understandings, skills, & affective qualities 
are required to do this version of the task? 

□ Let’s analyze this knowledge in terms of what people will have to learn 
to do the task. 

5. Adjust episode size. 

□ How much time should one class/learning session last? 

□ Do you think this is the right amount of material for one session? 

□ How much should we include? 

6. Design the within-episode sequence. 

□ Is this material in the order it will be used? 

□ Have we left any prerequisites out or gotten them out of order? 

□ Is there anything else they need to know before trying any parts of the 
task? 

□ Does the material seem to be grouped together well? 

Phase III - Identify the next learning episode. 

7. Identify the next simplest version. 

□ Let’s rank the simplifying conditions we have in order of complexity. 

□ What do you think might be the next simplest version of the task? 

□ What do you think would be the best simplifying conditions for this 
version of the task? 

8. Repeat steps 3-7 to identify all remaining versions of the task. 
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